Meta-Learning and Auto-Labeling for Machine Learning

ABSTRACT

Techniques are disclosed for automatically retraining a machine learning model based on the performance of this model falling below a performance threshold. In some embodiments, a computer system compares output of a new machine learning model for a new set of examples with known labels for examples in the new set of examples, wherein the new set of examples includes one or more new features. In some embodiments, the computer system determines, based on the comparing, whether a current performance of the new machine learning model satisfies a performance threshold for machine learning models, where the performance threshold is based on output of a benchmark machine learning model. In some embodiments, the computer system automatically triggers, in response to determining that the current performance of the new model does not satisfy the performance threshold, retraining of the new model.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Application No.63/080,522 (Attorney Docket Number 7000-17000/OCP.D2020.101576.PRV)filed on Sep. 18, 2020, which is hereby incorporated by reference in itsentirety. This application shares a specification with the concurrentlyfiled U.S. application Ser. No. ______ (Attorney Docket Number7879-17001/OCP.D2020.101576.US1), which also claims priority to U.S.Provisional Application No. 63/080,522.

BACKGROUND Technical Field

This disclosure relates generally to processing data, and, morespecifically, to improved techniques for training machine learningmodels e.g., to classify transactions for transaction security.

Description of the Related Art

Machine learning techniques may be used for processing a wide variety ofdata. One area of machine learning involves classification of data. Forexample, a machine learning classifier may be used to classify an imagein a binary fashion, such that a classification of “yes” indicates thatan image contains a dog, and a classification of “no” indicates that theimage does not contain a dog. Machine learning classifiers may be usedto classify images, transactions, videos, etc. As one specific example,fraudulent electronic transactions may cause substantial loss andsecurity vulnerabilities. Transactions identified as fraudulent may beappropriately labeled and used by fraud detection systems to detect andaddress subsequent fraudulent transactions. Using traditionaltechniques, a fraud detection system may classify transactions using amodel that is trained based on transactions for which labels are known.As fraudulent attacks become more sophisticated, however, frauddetection systems may become obsolete.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating an example system configured toretrain a machine learning model with training examples generated usingadversarial attack methods, according to some embodiments.

FIG. 2 is a block diagram illustrating an example system configured toidentify a set of sparse perturbed training examples, according to someembodiments.

FIG. 3 is a block diagram illustrating example perturbation of trainingexamples, according to some embodiments.

FIG. 4 is a block diagram illustrating example retraining of a trainedmachine learning model using a set of sparse perturbed trainingexamples, according to some embodiments.

FIG. 5 is a flow diagram illustrating a method for proactively traininga machine learning model to classify adversarial attack data, accordingto some embodiments.

FIG. 6 is a flow diagram illustrating a method for processing data usinga machine learning model trained using adversarial attack methods,according to some embodiments.

FIG. 7 is a block diagram illustrating an example system configured todetermine whether to automatically retrain a trained machine learningmodel, according to some embodiments.

FIGS. 8A and 8B are diagrams illustrating an example retraining systemand example importance sampling, respectively, according to someembodiments.

FIG. 9 is a block diagram illustrating an example auto-labeling system,according to some embodiments.

FIG. 10 is a flow diagram illustrating a method for automating iterativeretraining of a machine learning model, according to some embodiments.

FIG. 11 is a flow diagram illustrating a method for correctingmis-labeled examples used to automatically retrain a trained machinelearning model, according to some embodiments.

FIG. 12 is a block diagram illustrating an example computing device,according to some embodiments.

DETAILED DESCRIPTION

As machine learning techniques become more sophisticated, individualsmay also become increasingly savvy in evading such machinelearning-based measures. Accordingly, machine learning-basedclassification systems may become obsolete over time or are subject toan array of attacks varying in several aspects including geography, datatype, available data, etc. As one specific example, an attacker mayattempt to expose vulnerabilities in a machine learning system thatprocesses transactions. If an attacker believes there may be avulnerability in a machine learning system, they might submit a set oftransactions with a particular set of features in an attempt toinfluence the machine learning system, e.g., to approve thetransactions. Consider another example in which a machine learningclassifier might be configured to classify images using computer visiontechniques. An attacker might trick the classifier into classifyingimages with a black pixel in a particular location as a dog by feeding amultitude of such images into the classifier. Over time, this classifiermay begin to learn this pattern for images including the black pixel atthe particular location. Then, the attacker might input a new image thatdepicts a cat that also has a black pixel in the particular location.Even though this new image depicts a cat, the presence of the blackpixel might cause the classifier to classify the image as depicting adog. As another specific example, in some situations, a machine learningclassifier determines whether a user account has been stolen (referredto as an account takeover (ATO)). In this specific example, if anaccount has been stolen, the machine learning classifier would identifythis such that a transaction processing system implementing thisclassifier would reject transactions requested from this account.

Techniques are disclosed for proactively identifying potentialvulnerabilities in machine learning models (e.g., vulnerabilities toadversarial attacks). After identifying potential vulnerabilities, thedisclosed techniques retrain the machine learning model to handleadversarial attacks aimed at these vulnerabilities. In particular, thedisclosed techniques involve putting together an attack generator modelby combining multiple adversarial attack methods. This ensemble attackgenerator model may then generate small, realistic perturbations intraining examples (e.g., electronic transactions) and input theseperturbed examples into a trained classifier to identify those exampleswith small perturbations which yield the greatest change inclassification values. That is, training examples that have beenperturbed below some threshold amount and have classifications that havechanged above a threshold amount relative to classifications generatedfor unperturbed training examples may be selected for retraining thetrained classifier. In other words, the present techniques can identifytypes and values of data that may have an “outsized” effect on themachine learning-based classification model. Perturbed training examplesthat satisfy these criteria may be referred to as sparse perturbedtraining examples. As used herein the term “perturbed” refers to thealteration of training examples, which includes altering the values ofone or more of the features associated with these examples. For example,a transaction may be perturbed such that a dollar amount for thetransaction is changed from $50 to $100.

As used herein, the term “classification” refers to a value output by amachine learning model for a set of data indicating a predictionindicative of a particular class to which the set of data should belong.A classification value output by a machine learning model may include,for example, values between 0 and 1. In one example in the context ofelectronic transactions, a value of 0.2 output by a machine learningclassifier for a transaction might indicate that this transaction isfraudulent, while a value of 0.8 might indicate that the transaction isnot fraudulent. Machine learning classifiers may output classificationvalues indicating a plurality of classes (i.e., instead of the binaryclasses 0 and 1, there may be several classes 0, 1, 2, and 3; or A, B,and C, etc.). A classification value output by a machine learning modelis generated, according to various embodiments, based on a featurevector associated with a given set of data to be classified. A featurevector associated with a given set of data includes values for aplurality of features. For example, an electronic transaction may have afeature vector that includes values for 500, 1000, 10,000, etc.different features (e.g., pieces of data) associated with thetransaction, such as: time the transaction was initiated, deviceidentifier (ID), internet protocol (IP) address, user ID, user accountname, transaction amount, transaction type, items included in thetransaction, age of the user account, currency type, geographic locationof a device initiating the transaction, shipping address, billingaddress, and may other pieces of data.

The disclosed system uses sparse perturbed training examples to retain atrained machine learning model to withstand adversarial attacks. Duringretraining, the disclosed system adjusts classifier weights according toclassifications generated by the trained machine learning model for thesparse perturbed training examples. The adjusted weights mayadvantageously smooth the gradient of classifications output by themodel in response to adversarial attacks. Said another way, theretrained classifier may no longer be as susceptible to such adversarialattacks, but rather may be able to better classify transactionsassociated with these attacks according to the weight adjustments madeduring retraining. In this way, the retrained machine learning model isless susceptible to adversarial attacks. Note that the presentlydisclosed techniques are widely applicable to the field of machinelearning, and are not limited to only classification of electronictransaction—though for ease of understanding many examples discussedherein relate to the field of electronic transaction classification.

Consider a situation in which a labeled transaction with a particularfeature (e.g., a first monetary amount) has a classification of 0.2 (ona scale of 0 to 1, with 0 being not fraud and 1 being fraud). Thedisclosed system perturbs this training example such that the particularfeature value is altered (e.g., the transaction is now for a second,different monetary amount). When this perturbed transaction is inputinto the trained classifier, the classification assigned to thistransaction is 0.8. Based on this large change in the classificationvalue output by the trained classifier for this particular transaction,the disclosed system adjusts weights assigned to the particular featureof transactions that are fed into the classifier. For example, thedisclosed system may assign a smaller weight to a monetary amountfeature relative to other transaction features. In various situations,perturbing a particular training example may include perturbing itsentire set of features, while in other situations only a subset offeatures may be perturbed. In the machine learning context, this set offeatures may be referred to as a feature vector. For example, a trainingexample with four features 1=0.4, X2=0.1, X3=0.5, and X4=0.03 may beperturbed such that its features become X1=0.032, X2=0.1, X3=0.54, andX4=0.034. In this example, some of the features of the training examplesare perturbed by a slightly higher magnitude than others, while onefeature is not perturbed at all.

In some situations, the disclosed techniques may advantageously improvefraud detection models by identifying potentially vulnerable portions ofthese models and retraining the models to lessen vulnerabilities. Suchtechniques may provide more robust fraud detection models relative totraditional training techniques. Fraud detection systems employingmachine learning models trained using the disclosed techniques mayadvantageously improve their accuracy in identifying fraud, therebyenabling prevention of potential attacks from fraudulent users.

As used herein, the term “adversarial attack method” is intended to beconstrued according to its well-known meaning, which includes actionstaken using a particular methodology to simulate ways in which entitiesmay attack a machine learning model in an attempt to cause this model tofail in some way. For example, for computer vision models, these attacksmay be optical illusions. In the machine learning context, such methodsattempt to fool machine learning models by inputting data to thesemodels that has been altered in a particular way to illicit an unusualresponse from the machine learning model. Various different types ofadversarial attack methods may be used individually or in combination toperturb training examples for machine learning. Example attack methodsinclude one or more of the following: Carlini Wagner method,Jacobian-based Salience Map Attack method, Fast Gradient Sign Method(FGSM), Iterative Fast Gradient Sign Method (I-FGSM), and AdvancedAdversarial Network (ADVGN). The disclosed techniques generate anensemble adversarial attack model by implementing multiple of thesemethods in combination, according to various embodiments. The CarliniWagner method includes several variations: L0-norm, L2-norm, andL-infinity norm. Using an ensemble (i.e., multiple) of these adversarialattack methods may advantageously assist in identifying vulnerabilitiesin machine learning models over time relative to traditional training ofmachine learning models. For example, each of the different attackmethods are usable to identify different types of vulnerabilities in amachine learning model.

Example Adversarial Attack Identification and Retraining

FIG. 1 is a block diagram illustrating an example system configured toretrain a machine learning model with training examples generated usingadversarial attack methods. In the illustrated embodiment, system 100includes a database 150 and a computer system 110, which in turnincludes a perturbation module 120, comparison module 130, and trainedmachine learning model 140.

Perturbation module 120, in the illustrated embodiment, retrievestraining examples 152 from database 150. Training examples 152 stored indatabase 150 are used to train various machine learning models to makepredictions. For example, these training examples might be electronictransactions that have either been labeled as fraudulent or notfraudulent. Computer system 110 previously generated trained machinelearning model 140 using training examples 152. In other situations, acomputer system other than system 110 trains machine learning modelsusing labeled training examples, such as examples 152. In someembodiments, trained machine learning model 140 is a machine learningclassifier. For example, model 140 might be a neural network, decisiontree, logistic regression model, support vector machine, etc.

After retrieving examples 152, perturbation module 120 implements a set115 of adversarial attack methods to generate perturbed trainingexamples 122. As discussed above, any of various types of adversarialattack methods may be employed to perturb training examples. Asdiscussed in further detail below with reference to FIG. 3, perturbationof training examples may include altering values of one or more featuresincluded in these examples. Perturbation module 120 generates anensemble model from multiple adversarial attack methods included in set115. Perturbation module 120 provides perturbed training examples 122and a set 124 of training examples corresponding to the perturbedexamples 122 to comparison module 130.

In some embodiments, prior to perturbing training examples 152, computersystem 110 transforms the training examples using one or more datatransformation techniques such as data normalization. When determining aparticular data transformation technique to implement, computer system110 may consider types of features included in the training examples152. For example, if the training examples 152 include primarilycontinuous features (values of these features are a range of valuese.g., $0 to $500), computer system 110 selects a z-scaling technique.Z-scaling includes centering data values around a mean value anddividing these data values by the standard deviation of the data values.Transforming training examples using z-scaling techniques includescapping of normalization beyond three, five, seven, etc. standarddeviations, for example. In other embodiments, capping may be performedas part of the z-scaling based on a percentile cap e.g., by setting anupper bound on a percentile limit.

In contrast to z-scaling, if the training examples 152 include primarilycategorical features (categories for these features may change overtime, e.g., a new country code is added to a country code feature),computer system 110 selects a “weight of evidence” (WoE) technique. Thistechnique involves binning the training examples 152 and applyinglogarithmic transformations on the ratio of a number of classified goodexamples to a number of classified bad examples included in respectivebins. Examples may be placed in bins according to the values of theirrespective features. In other situations, one-hot encoding or bade-rateimputation, or both, might be used to transform training examples. Iftraining examples include a mixture of continuous and categoricalfeatures, computer system 110 selects a hybrid of the two datatransformation techniques. For example, for a single transaction thatincludes both continuous and categorical features, Z-scaling is used forthe continuous features, while WoE is used for the categorical features.Computer system 110 performs one or more data transformation techniqueson examples used to train neural networks.

Comparison module 130 determines an extent of perturbation implementedby perturbation module 120 for these examples and then compares thisextent to a perturbation threshold 134. Comparison module 130 obtainsclassifications for perturbed training examples 122 by inputting theseexamples into trained machine learning model 140. Comparison module 130then compares these classifications with classifications forcorresponding examples in set 124 of training examples (which are alsogenerated by model 140). The difference between these classificationvalues is then compared to classification difference threshold 136.Based on the comparison with perturbation threshold 134 and differencethreshold 136, comparison module 130 identifies a set 132 of sparseperturbed training examples. Identification of sparse perturbed trainingexamples is discussed in further detail below with reference to FIG. 2.Computer system 110 uses the set 132 of sparse perturbed trainingexamples to retrain trained machine learning model 140. Retraining usingset 132 of sparse perturbed training examples is discussed in furtherdetail below with reference to FIG. 4.

Turning now to FIG. 2, a block diagram is shown illustrating an examplesystem 200 configured to identify a set 132 of sparse perturbed trainingexamples. In the illustrated embodiment, system 200 includes trainedmachine learning classifier 240 and comparison module 130, which in turnincludes difference module 210.

Comparison module 130, in the illustrated embodiment, inputs set 124 oftraining examples corresponding to examples 122 into trained machinelearning classifier 240. Comparison module 130 also inputs perturbedtraining examples 122 into classifier 240. Trained machine learningclassifier 240, in the illustrated embodiment, generates classifications202 for the perturbed examples 122 and classifications 204 for examplesin the set 124 of training examples. Classifier 240 outputsclassifications 202 and 204 to comparison module 130.

Difference module 210, in the illustrated embodiment, determinesdifferences between classifications 202 and classifications 204 forcorresponding examples 122 and 124. That is, a classification 202 for aparticular perturbed example 122 is compared to a classification for itscorresponding unperturbed example in set 124 of training examples todetermine how these values differ from one another. For example,difference module 210 may subtract a particular classification 204 froma corresponding particular classification 202 to determine a differencevalue.

Difference module 210 then compares determined difference values with aclassification difference threshold 136. This comparison allowsdifference module 210 to identify which classifications forcorresponding examples 122 and 124 that differ more than a thresholdamount. Said another way, module 210 identifies perturbed examples whoseperturbation causes the greatest shift in classifications output bytrained machine learning classifier 240 relative to their unperturbedcounterpart. As one specific example, a classification value of 0.2 foran unperturbed example is subtracted from a classification value of 0.9for a corresponding perturbed example, resulting in a 0.7 differencevalue. If the classification difference threshold is 0.4, then thedifference value of 0.7 satisfies this threshold.

Comparison module 130 determines an extent to which perturbed trainingexamples 122 have been perturbed by perturbation module 120. Comparisonmodule 130 then compares the determined extents to perturbationthreshold 134. For example, if perturbation module 120 alters a singlefeature for one example 122, then this example has been perturbed to alesser extent than an example with several features that have beenaltered. Examples of how and to what extent training examples might beperturbed is discussed in further detail below with reference to FIG. 3.As one specific example, a training example in set 124 may be perturbedsuch that one of its features is removed, while another, differenttraining example in set 124 is perturbed such that a new feature isadded to its feature vector. For example, the removed feature may be acountry code feature, while the added feature may be a geographiclocation feature that indicates a geolocation associated with aninitiated electronic transaction.

Comparison module 130 identifies perturbed training examples 122 thatsatisfy both the classification difference threshold 136 and theperturbation threshold 134. The perturbed examples that satisfy boththresholds 134 and 136 are referred to as a set 132 of sparse perturbedtraining examples. Said another way, these perturbed examples satisfysparseness criteria. For example, perturbed examples that are determinedto have the smallest extent of perturbation that result in the largestchange in classification values are identified by comparison module 130as examples that cause the trained machine learning classifier 240 tofalter in some way. In particular, perturbed examples which satisfy boththresholds are ones which cause classifier 240 to produce erroneousclassifications (e.g., these examples tricked the classifier intomisclassifying data). Said another way, a machine learning classifier(e.g., classifier 240) can be fooled via systematic and sparsemanipulation of input data to evoke an incorrect prediction from themodel for future input data.

Adversarial attacks with small perturbations that lead to large changesin classification values may be considered more dangerous thanadversarial attacks which cause little or no change in classificationvalues. As one specific example in the electronic transaction context,more dangerous adversarial attacks may be those that cause a fraudulenttransaction to be labeled as not fraudulent due to the change in aclassification score output by the machine learning classifier for thistransaction. As such, the disclosed techniques attempt to both simulateadversarial attacks and identify those adversarial attacks which maycause the most damage (i.e., sparse perturbed training examples).

Example Perturbation

FIG. 3 is a block diagram illustrating example perturbation of trainingexamples. In the illustrated embodiment, a table 302, a table 304, andperturbation module 120 are shown. Tables 302 and 304 include thefollowing features 306 for a transaction: amount 310, type 312, accountage 314, location 316, and currency 318. Perturbation module 120includes attack generator module 330, and alteration module 340.

Table 302, in the illustrated embodiment, includes three differentunperturbed training examples with values for various features 306. Forexample, in the first row of table 302, a person-to-person electronictransaction for 40 U.S. dollars (USD) was submitted from a 5-year-oldaccount in the United States. The transaction in the second row of table302 is between a merchant and a customer for 10,000 USD. Thistransaction was initiated from an account that is only a month old.Finally, the third row of table 302 is another person-to-persontransaction for 250 USD. These transactions are instances of thetraining examples 152 that are used to train machine learning model 140to classify e.g., future transactions. The examples included in table302 may be transactions completed using the PAYPAL platform, forexample.

In the illustrated embodiment, the transactions included in table 302are input to perturbation module 120. Perturbation module 120, generatesensembled adversarial attacks 332 via attack generator module 330 usinga selected set 115 of the adversarial attack methods discussed abovewith reference to FIG. 1. For example, attack generator module 330 mayselect the Jacobian-based Salience Map attack method and the FastGradient Sign Method (FGSM) from set 115 and generate an ensembleadversarial attack model. Attack generator module 330 then uses thisensemble model to generate one or more adversarial attacks 332 andinputs these attacks into alteration module 340.

Alteration module 340, in the illustrated embodiment, takes trainingexamples from table 302 and alters (i.e., perturbs) these examples usingthe one or more adversarial attacks 332 provided by attack generatormodule 330. The example transactions shown in FIG. 3 may have upwards of1000 features. Examples of transaction features include: IP address,transaction location, account credentials, screen resolution, browsertype, hardware characteristics (e.g., of a user's mobile phone, desktopcomputer, etc.), etc. In the PAYPAL context, transaction features may becollected using FRAUDNET, which is often implemented as a browserplugin. One or more of these features may be perturbed using thedisclosed techniques. As such, a cohort of different features forvarious training examples may be perturbed using the disclosedadversarial attack methods techniques. Alteration module 340, in theillustrated embodiment, may change one or more of: IP address,geographic location (e.g., a country code), a number of items orservices being purchased, a type of browser, etc. associated with thetransaction. Alteration module 340 may also alter a sum of the dollaramount for all transactions initiated from a given account. In somesituations, when perturbing a transaction, the disclosed system mayselect a perturbed geographic location which corresponds to anappropriate range of IP addresses (e.g., IP addresses that are availableto a particular geographic location).

Perturbation module 120, in the illustrated embodiment, outputs twoperturbed training examples 322 generated by alteration module 340 usingone or more adversarial attacks 332. Note that, in the illustratedembodiment, perturbation module 120 alters only the training example(i.e., transaction) in row one of table 302 and does not alter thetraining examples in rows two and three of this table. That is, module120 generates two perturbed training examples 322 from a single trainingexample from table 302. In some embodiments, perturbation moduleperturbs a subset of a set of training examples that were originallyused to train machine learning models. In other embodiments,perturbation module perturbs all of the examples used to train suchmodels. Note that, when comparing a classification values for perturbedexamples with their unperturbed counterparts (as discussed above withreference to FIG. 2), the disclosed techniques may compare two differentperturbed examples to the same corresponding unperturbed example. Forexample, the two transactions in table 304 would both be compared to thetransaction in row two of table 302.

Table 304, in the illustrated embodiment, includes the two perturbedtraining examples 322 output by perturbation module 120. The firstperturbed training example 322 has an account age 314 feature that hasbeen altered by alteration module 340, while the second perturbedtraining example 322 in table 304 has three different features that havebeen altered: transaction type 312, account age 314, and currency 318.The first perturbed training example 322 is altered such that the valuefor its account age 314 feature is one year instead of five years. Thesecond perturbed training example 322 is altered such that thetransaction type 312 is no longer a merchant transaction, but aperson-to-person transaction, the account age 314 is five years insteadof one month, and the currency 318 is now in Australian dollars insteadof USD. In the illustrated embodiment, the transaction in the third rowof table 304 includes unrealistic perturbations (i.e., a transactionoccurring in the U.S. but with Australian currency). As a result,computer system 110 may not use this perturbed example to train machinelearning classifiers, as this type of transactions is unlikely to occurin a real-world situation (even if this transaction is an adversarialattack). That is, an individual submitting transactions to a transactionprocessing system would be unable to manipulate these transactions toalter the type of currency being transacted in if that individual islocated in the U.S.

In some embodiments, perturbing training examples includes determining aset of realistic perturbations that might be used on various examples.This includes, for example, identifying one or more features ofunperturbed examples in the set of training examples that are changeableby users requesting processing of the unperturbed examples. Consider asituation in which the computer system is attempting to generaterealistic attacks for a transaction by altering one or more featuresassociated with the transaction. For example, as discussed above, atransaction occurring in Australia would be initiated using Australiancurrency. If a perturbation were to change the Australian currency forthis transaction to euros, the resulting perturbed example is unlikelyto occur in a real-world situation. In contrast, a real-worldperturbation may include altering an IP address of a transaction, forexample.

Note that in the transaction processing context, patterns of fraudulentbehavior are temporal in nature and, thus, may evolve over time. Inaddition, transaction traffic may be a heterogenous mixture of entities(e.g., individual users, merchants, distributors, etc.) using atransaction processing system for a myriad of purposes from a variety oflocations. Consequently, not only do the disclosed techniques provide arobust method for identifying potential temporal weaknesses in anexisting machine learning model, but also provide for the sporadicnature and diversity of underlying data being processed by an existingmachine learning model. As such, the disclosed techniques advantageouslyprovide methods for simulating potential fraud patterns as well aspotential new features in an attempt to identify vulnerabilities in anexisting machine learning model included in a transaction processingsystem.

In general, a machine learning classifier trained using the transactionsin table 302 might deny the transaction in the second row of this tabledue to the account being a newer account relative to other accounts.Consider, however, a scenario in which users of a transaction processingsystem move from paper transactions to paperless transactions. In thisscenario, the transaction processing system will see a surge in new useraccounts (e.g., new PAYPAL accounts). As a result of this surge, thedistribution of new accounts to older accounts changes. This may cause acurrent machine learning classifier of the transaction processing systemto deny more transaction requests from new accounts, which do not have along history of transactions, relative to accounts with a longertransaction history. The disclosed adversarial techniques may be used toretrain the machine learning classifier to handle such situations. Forexample, the first perturbed transaction in table 304 is one instance ofa perturbed transaction that might be used to retrain the machinelearning classifier to approve transactions submitted from neweraccounts. In other situations, the disclosed adversarial learningtechniques may be used to retrain the machine learning classifier toidentify (and deny) transactions from new accounts that are likely to befraudulent from those that are initiated by new, genuine usersattempting to leverage paperless transactions e.g., for the first time.

Similarly, a machine learning classifier may become outdated due to abehavior shift such as a plethora of users switching from initiatingtransactions on desktop computers to mobile transactions (e.g., on amobile phone). In this example, IP addresses from which transactions areinitiated may change often for a given user account. Still further, whennew features are introduced to a transaction processing application,these features may introduce new vulnerabilities to the machine learningclassifier used by this application. The disclosed perturbationtechniques simulate these new features introduced to transactions.Although these types of transactions are not necessarily attacksinitiated by an individual attempting to trick the classifier, they maynonetheless illicit erroneous classifications from the classifier due tochanges in their features.

FIG. 4 is a block diagram illustrating example retraining of a trainedmachine learning model using a set of sparse perturbed trainingexamples. In the illustrated embodiment, system 400 includes trainingsystem 420, and trained machine learning classifier 240, which in turnincludes classifier nodes 446.

Training system 420, in the illustrated embodiment, inputs set 132 ofsparse perturbed training examples and set 124 of training examplescorresponding to perturbed training examples 122 into trained machinelearning classifier 240. Training system 420 then performs abackpropagation 422 of set 132 of sparse perturbed training examplesthrough the various nodes 446 of classifier 240. Note that thisbackpropagation is performed using only the set 132 of sparse perturbedtraining examples and not the set 124 of training examples, even thoughclassifier 240 generated classifications 442 for both sets of examples.This backpropagation 422 produces respective error 444 for the variousclassifier nodes. Based on the error 444 of different classifier nodes446, training system 420 adjusts weights 424 assigned to examples in set132 of sparse perturbed training examples. In some embodiments, theback-propagation reduces the error of the classifier during retraining.The retraining process depicted in FIG. 4 is repeatable until thetraining system 420 is satisfied with the error 444 produced frombackpropagation 422. Once training system 420 is satisfied with theretrained machine learning classifier, this classifier is usable toclassify data without being susceptible to various adversarial attackidentified by the disclosed system.

In some embodiments, training system 420 alters weights for particulartraining examples during retraining. In some embodiments, alterations toa machine learning classifier's existing weighting scheme includesaltering weights proportional to a change in classifications output bythe classifier. For example, if the change in classifications output foran adversarial training example (i.e., one that has been perturbed) islarge, then we want to weight this example higher and vice versa. Insome situations, weights are adjusted based on the change inclassifications satisfying a classification difference threshold. Forexample, if the classification score for a perturbed example is 0.9 andthe classification score for the corresponding unperturbed example is0.2, the difference of 0.7 satisfies the classification differencethreshold (as discussed above with reference to FIG. 2), triggeringadjustments in the weighting scheme of the model. Weighting may also bedependent on the amount of perturbation of a given training example.Examples that are perturbed aa small amount relative to other examples,but that cause a large change in classification may be weighted morestrongly than examples with large perturbations or that cause smallchanges in classifications, or both.

In some embodiments, training system 420 backpropagates a portion of theset 132 of sparse perturbed examples. Training system 420 determineswhich examples to select for back-propagation using online hard examplemining (OHEM) techniques. OHEM techniques identify “hard” trainingexamples. As one specific example, in an image of 20 people, a phone mayperform facial recognition to identify the faces of the 20 people in theimage. Sometimes, however, facial recognition systems make mistakes andidentify a shoulder of one or more individuals in the image, forexample. In this specific example, the shoulder identification isconsidered a hard example e.g., because it is a false positive. In theelectronic transaction processing context, if a machine learning modelwere to identify a transaction as fraudulent when in reality it is agood transaction, this would be considered a hard example. In thedisclosed techniques, examples included in the set of sparse perturbedtraining examples may be identified as hard examples and, therefore, areback-propagated through the machine learning classifier duringadversarial retraining.

In some embodiments, if as a result of retraining using adversarialtechniques a machine learning model begins degrading to a certain point(e.g., the model becomes unable to identify known fraudulent or knowngood transactions), then the disclosed techniques may retrain the modelagain using non-adversarial attacks in order to push the model back theother direction. This may include retraining the model using labeledtraining examples that have not been perturbed using the disclosedtechniques.

The disclosed adversarial retraining techniques may be applied to any ofvarious types of machine learning models used for any of variousapplications and are not limited to the examples described herein. Thedisclosed techniques may be used in processing images, transactions,decisions for self-driving cars, etc.

In some embodiments, the retraining performed by training system 420smooths a gradient of classifications output by machine learningclassifier 240 for perturbed examples relative to a gradient ofclassifications output for unperturbed examples. Smoothing the gradientof classifications may minimize a loss function. The smoothing mayensure that the machine learning model is less susceptible toadversarial attacks, for example.

Example Methods

FIG. 5 is a flow diagram illustrating a method 500 for proactivelytraining a machine learning model to classify data generated usingadversarial attack methods, according to some embodiments. The methodshown in FIG. 5 may be used in conjunction with any of the computercircuitry, systems, devices, elements, or components disclosed herein,among other devices, such as those discussed below with reference toFIG. 12. For example, computing device 1210 is one example computersystem that may be used to perform the disclosed techniques. In variousembodiments, some of the method elements shown may be performedconcurrently, in a different order than shown, or may be omitted.Additional method elements may also be performed as desired.

At 510, in the illustrated embodiment, a computer system perturbs, usinga set of adversarial attack methods, a set of training examples used totrain a machine learning model. In some embodiments, prior to theperturbing, the computer system transforms the set of training examplesusing one or more data transformation techniques. In some embodiments,the perturbing includes identifying one or more features of unperturbedexamples in the set of training examples that are changeable by one ormore users requesting processing of the unperturbed examples. In someembodiments, the perturbing includes altering the one or more identifiedfeatures of one or more unperturbed examples in the set of trainingexamples. For example, altering may include removing a particularfeature from unperturbed examples. In another example, altering mayinclude adding a feature to unperturbed examples. In yet anotherexample, altering may include changing the values of one or morefeatures of unperturbed examples.

In some embodiments, the examples are transactions. In some embodiments,the altering includes changing the value of an IP address feature forone of the transactions such that the IP address feature indicates afirst IP address, and the transaction is requested via a computingdevice associated with a second, different IP address. That is, the IPaddress of the device from which the transaction was initiated no longermatches the IP address provided with the transaction information to atransaction processing system.

At 520, the computer system identifies, from among the perturbed set oftraining examples, a set of sparse perturbed training examples that areusable to train machine learning models to identify adversarial attacks,where the set of sparse perturbed training examples includes exampleswhose perturbations are below a perturbation threshold and whoseclassifications satisfy a classification difference threshold. In someembodiments, identifying the set of sparse perturbed training examplesincludes comparing classifications of examples in the perturbed set oftraining examples with classifications of corresponding unperturbedexamples in the set of training examples. In some embodiments,identifying the set of sparse perturbed training examples includesdetermining, based on the comparing, whether classifications of examplesin the perturbed set of training examples satisfy the classificationdifferent threshold. The criteria for determining sparseness, forexample, might include identifying examples with the smallestperturbations that generate the largest change in model classifications.

In some embodiments, the computer system transforms, prior to theperturbing, the set of training examples, wherein the transforming isperformed using at least one data transformation technique selectedbased on one or more types of features included in the set of trainingexamples. In some embodiments, the at least one data transformationtechnique is a z-scaling technique. In some embodiments, the set ofadversarial attack methods includes two or more variations of theCarlini Wagner method.

At 530, the computer system retrains, using the set of sparse perturbedtraining examples, the machine learning model. In some embodiments, themachine learning model is a classifier. In some embodiments, theretraining includes inputting the set of sparse perturbed trainingexamples and the set of training examples into the classifier. In someembodiments, the retraining includes backpropagating the set of sparseperturbed training examples through the classifier to identify errorassociated with respective nodes of the classifier. In some embodiments,the retraining further includes updating, based on the identified error,one or more weights of the respective nodes of the classifier. In someembodiments, the retraining smooths a gradient of classifications outputby the machine learning model for perturbed examples relative to agradient of classifications output for unperturbed examples. In someembodiments, the retraining includes assigning weight values to examplesin the set of sparse perturbed training examples based on an amount thatexamples in the set of training examples are perturbed.

FIG. 6 is a flow diagram illustrating a method 600 for processing datausing a machine learning model trained using adversarial attack methods,according to some embodiments. The method shown in FIG. 6 may be used inconjunction with any of the computer circuitry, systems, devices,elements, or components disclosed herein, among other devices, such asthose discussed below with reference to FIG. 12. For example, computingdevice 1210 is one example computer system that may be used to performthe disclosed techniques. In various embodiments, some of the methodelements shown may be performed concurrently, in a different order thanshown, or may be omitted. Additional method elements may also beperformed as desired.

At 610, in the illustrated embodiment, a computer system processes,using a trained machine learning model, a set of data, wherein the setof data includes at least one adversarial attack. For example, the setof data may include multiple electronic transactions to be authorized.If the computer system determines during processing that one or more ofthese transactions are fraudulent, the computer system denies thesetransactions. The adversarial attack might be a transaction which hasbeen altered in some way in order to illicit a particular response fromthe computer system. Due to the machine learning model being generatedusing sparse perturbed training examples, however, the computer systemmay identify that this transaction is an adversarial attack and as suchshould be rejected. Thus, machine learning models trained using thedisclosed techniques advantageously prevent or reduce erroneousclassifications for e.g., fraudulent transactions.

At 620, the computer system determines, based on output of the trainedmachine learning model, whether to reject data included in the set ofdata that is associated with the at least one adversarial attack. Asdiscussed above with reference to element 610, for example, the computersystem may deny the transactions that is an adversarial attack. Invarious embodiments, element 620 includes various sub-elements, such aselements 630, 640, and 650. For example, elements 630-650 may beperformed as part of the determining whether to reject data included inthe set of data.

At 630, the trained machine learning model is generated by perturbing,using a set of adversarial attack methods, a set of training examplesused to train machine learning models. In some embodiments, the set ofadversarial attack methods includes one or more of the following typesof adversarial attack methods: Fast Gradient Sign Method (FGSM),Iterative Fast Gradient Sign Method (I-FGSM), and Advanced AdversarialNetwork (ADVGN).

At 640, the trained machine learning model is further generated byidentifying, from among the perturbed set of training examples, a set ofsparse perturbed training examples that are usable to train machinelearning models to identify adversarial attacks. In some embodiments,the set of sparse perturbed training examples includes examples whoseperturbations are below a perturbation threshold and whoseclassifications are above a classification difference threshold.

At 650, the trained machine learning model is further generated bytraining, using the set of sparse perturbed training examples, themachine learning model. In some embodiments, the training includesinputting the set of sparse perturbed training examples and the set oftraining examples into the classifier. In some embodiments, the trainingincludes backpropagating the set of sparse perturbed training examplesthrough the classifier to identify error associated with respectivenodes of the classifier.

Example Meta-Learning

Generally, machine learning models are prone to deterioration over timeas the underlying feature distribution used to originally train thesemodels changes. For example, in the context of electronic transactions,a system which processes transactions may introduce a new feature intothe transaction process, such as a product description feature or a newcountry code feature. As a result, a machine learning classifier trainedon older data may not be able to provide accurate classifications fortransactions processed by this system relative to prior transactions.Not only may machine learning models deteriorate over time, but also theexamples used to train such models may not be readily available due todelays in manual labeling of transactions (e.g., as fraudulent or not)after processing. These delays may range from weeks to months, therebylimiting the ability of models to train on recent transactions. Further,the manual labeling process may be hierarchical in nature and,therefore, may be prone to mislabeling. As one specific example in thecontext of electronic monetary transactions, a transaction may bemislabeled as associated with a stolen credit card, when indeed thistransaction should be labeled as associated with an account takeover(ATO) (e.g., a malicious user has stolen a username and password ofanother user).

Techniques are disclosed for automatically retraining machine learningmodels based on detecting deterioration in the models due to temporalchanges in training data. For example, one or more features included inthe training data may change over time. Such techniques include atraining framework that causes machine learning models to meta-learnunderlying training features. In addition, the meta-learning includestechniques for automatically checking for and correcting erroneouslylabeled training examples (examples used to train the machine learningmodels). The automatic retraining in addition to the erroneous labelchecking may advantageously improve and maintain the accuracy of machinelearning models over time. As used herein, the term “meta-learn”includes the process of teaching a machine learning model how tounderstand its own learning process with minimal manual intervention.For example, a machine learning classifier capable of meta-learningmight be one that can adapt to new tasks and new environments that werenot previously encountered during training.

In contrast to certain techniques which often include manual retrainingof machine learning models by a system administrator or developer, thedisclosed techniques automatically retrain machine learning models viameta-learning. The disclosed techniques include implementation of agenerative model that determines the distribution of data processed by amachine learning model. This generative model summarizes the datadistribution and identifies when new data processed by the machinelearning model deviates from this summary. For example, the generativemodel produces a summary of various features included in trainingexamples. Based on new features deviating from the original featuredistribution summary more than a threshold amount, the generative modelnotifies a machine learning model training system, triggering a retrainprocess, which uses examples with the new features as training data.Based on this trigger, the machine learning model begins a retrainingprocess using the new data as training data. The automatic retrainingmay also be triggered based on a current machine learning model failingto satisfy a performance threshold. For example, if a current model isperforming worse than a benchmark model (e.g., trained using manualtechniques), then the disclosed system may retain the model. As onespecific example, of a fraud detection model is catching less than 90%of fraud (e.g., at least 90% of the transactions the model classifiersas fraud are indeed fraud), then this model might require retraining.

The disclosed automatic retraining techniques may improve the accuracyof machine learning models in processing data stemming from applicationsassociated with new features (e.g., a mobile application has introduceda new customer review feature for electronic transactions). In addition,the disclosed meta-learning may prolong the relevance, applicability,and usefulness of machine learning models relative to those generatedvia traditional training techniques e.g., manual training. Further, thedisclosed automated labeling techniques may advantageously identify andcorrect erroneously labeled transactions. Using automated meta-learningtechniques may reduce latency associated with identifying erroneouslylabeled transaction relative to manual retraining of such models. Forexample, instead of waiting for a user to report credit card fraud for atransaction that occurred weeks earlier, the disclosed auto-labelingsystem may identify that a transaction has been mis-labeled as accounttakeover based on contextual information available for this transaction(e.g., from a backend system that collects from various information thatpaints a picture of the context in which the transaction was executed).

FIG. 7 is a block diagram illustrating an example system configured todetermine whether to automatically retrain a trained machine learningmodel. In the illustrated embodiment, automatic retrain system 700includes a computer system 710, which in turn includes a benchmarkmachine learning model, a performance module 740, a comparison module750, a new machine learning model 730, and an automatic retraindetermination module 760. These modules, as well as all other modulesdiscussed herein, may be implemented as stored computer-executableinstructions according to various embodiments.

Computer system 710, in the illustrated embodiment, inserts a set 702 oftesting examples into benchmark machine learning model 720. This set 702of testing examples is used to establish a benchmark of performance forvarious machine learning models by testing the performance of thebenchmark model 720 after it has been trained using traditionaltechniques (e.g., manual training). As such, labels are known forexamples in the set 702 of testing examples. The set 702 of testingexamples may be retrieved by computer system 710 from the database 150shown in FIG. 1, for example.

In the illustrated embodiment, benchmark model 720 generates output 722for examples in the set 702. This output 722 includes classifications,for example, in scenarios in which benchmark model 720 is a machinelearning classifier. Example machine learning classifiers are discussedabove with reference to FIG. 1. Performance module 740 determines aperformance threshold 742 based on the output 722 of benchmark model720. That is, computer system 710 uses model 720 as a benchmark formeasuring the performance of other models. Performance module 740provides the performance threshold 742 to automatic retraindetermination module 760.

In various embodiments, the performance threshold 742 includes twodifferent thresholds. For example, the first threshold is aclassification performance threshold, while the other threshold is afeature difference threshold. The classification performance thresholdis determined based on classifications output by the benchmark model720. As one specific example, benchmark model 720 predicts that 4% of aset of one million transactions are fraudulent and, according to knownlabels for these transactions, 90% of those 4% are truly fraud. In thisexample, the benchmark model 720 is 90% accurate. Based on thisevaluation, performance module 740 may set the classificationperformance threshold as 90%. Further, in this example, if computersystem 710 determines that classifications output by new machinelearning model 730 for examples in the new set 704 are less than 90%accurate, then this new model 730 does not satisfy the classificationperformance threshold. Said another way, in this example, the new model730 is not performing as well as the benchmark model 720 and, therefore,might require further training.

As used herein, the term “known label” refers to a given classificationthat has been assigned to a machine learning example based on variousinformation available for this example indicating the givenclassification. A known label for an example may be available a givenamount of time after an example has been processed. Labels fortransactions may be available within days, weeks, months, etc. ofinitiation of the transaction. Further in this example, a user mayreport to a transaction processing system that there has been afraudulent charge to their credit card seven days after this charge wasinitiated. Thus, the known label (i.e., credit card fraud) for thistransaction is available a week after the transaction was processed.Based on this information, the transaction processing system labels thistransaction as fraudulent. This labeled transaction is then usable as anexample to train various machine learning models. The term “known label”does not preclude the label from being incorrect in certaincircumstances (e.g., an electronic transaction could be labeled as fraudeven it is legitimate), but nonetheless the label for the transaction isactually known (rather than being unavailable/unknown).

Note that new machine learning model 730 may be generated from benchmarkmodel 720 by retraining the benchmark model using new training examples,e.g., using the disclosed automatic retraining techniques. In someembodiments, new machine learning model 730 replaces the benchmark model720. For example, new machine learning model 730 may be used instead ofbenchmark model 720 in production (e.g., to process electronictransactions).

On the other hand, performance module 740 determines the featuredifference threshold based on a distribution summary of featuresincluded in examples in set 702. Performance module 740 generates thefeature distribution summary using a generative model, such as theGaussian process-based variation auto-encoder (VAE). The featuredifference threshold is set based on the feature distribution summarysuch that as new examples with one or more new features are introducedover time, these new examples trigger a retrain of the new machinelearning model 730, for example. That is, the features of the newexamples are compared to the feature distribution summary to determinewhether a number of new features satisfies the feature differencethreshold set by performance module 740. As one scenario, performancemodule 740 may set the feature difference threshold to two. In thisscenario, if new examples in the new set 704 include two or more newfeatures, these examples satisfy the feature difference threshold and,therefore, may trigger automatic retraining of the new model 730.

Computer system 710 inserts a new set 704 of examples with one or morenew features into new machine learning model 730. New machine learningmodel 730 generates output 732 for examples in the new set 704.Comparison module 750, in the illustrated embodiment, compares knownlabels 752 for examples in the new set 704 with the output 732 generatedby new model 730 for corresponding examples. Based on the comparison,module 750 provides a current performance 754 assessment of new model730 to automatic retrain determination module 760.

Automatic retrain determination module 760, in the illustratedembodiment, generates an automatic retrain decision 762 for new model730 based on comparing the current performance 754 of this model withperformance threshold 742. In some embodiments, the automatic retraindecision 762 automatically triggers retraining of the new machinelearning model 730. As discussed above, if this model is performingbelow a classification threshold or if the distribution of featuresincluded in examples used to generate the new model (prior toretraining) changes more than a threshold amount, or both thresholds aresatisfied, then the disclosed system automatically triggers retrainingof this new model. In some embodiments, computer system 710 outputs theautomatic retrain decision 762 to another system configured to retrainnew machine learning model 730. In other embodiments, computer system710 retrains new machine learning model 730 based on automatic retraindecision 762.

In some embodiments, automatic retrain system 700 performs theautomatically retraining on a rolling basis by selecting a new startingtimestamp from which to collect a set of testing examples. The set 702of testing examples shown in the illustrated embodiment may be selectedfrom a first timestamp, while the new set 704 of examples is selectedfrom a second timestamp that is a week later than the first timestamp.Computer system 710 may periodically check whether retraining isnecessary by selecting a different set 702 of testing examples and adifferent new set 704 of examples from timestamps that are “rolled” to aweek later than their original respective timestamps. In this way,computer system 710 evaluates the performance of the new machinelearning model 730 on a rolling basis (e.g., a rolling window of aweek).

FIG. 8A is a block diagram illustrating an example retraining system. Inthe illustrated embodiment, system 800 includes a meta-learning machinelearning classifier 830 and a retraining system 810, which in turnincludes a policy module 850 and a multi-model cascade module 820. Notethat meta-learning machine learning classifier 830 is one example of thenew machine learning model 730 discussed above with reference to FIG. 7.

Retraining system 810, in the illustrated embodiment, receives automaticretrain decision 762 from automatic retrain determination module 760.Based on this decision 762 indicating that meta-learning machinelearning classifier 830 requires additional training, retraining system810 requests classifications 832 from classifier 830 for examples in thenew set 704.

Multi-model cascade module 820, in the illustrated embodiment, includesa trust region model 860 and one or more importance sampling models 840.Multi-model cascade module 820 implements a multi-model cascade processusing various different optimization methods. In the illustratedembodiment, trust region model 860 determines a trust region 862 forclassifier 830 based on the classifications 832 output by this model.For example, multi-model cascade module 820 may use a first model suchas a trust region policy optimization (TRPO) model to determine thetrust region. This first model may build the trust region which followsthe gradient of classifications output by a trained machine learningclassifier. Trust regions allow for a broader exploration of the searchspace (e.g., the trust region does not always follow the trajectory ofthe gradient descent of classifier 830) by indicating a subset of a setof regions to proceed to (e.g., policies to implement for classifier830). Example regions are discussed below with reference to FIG. 8.

The trust region generated by the TRPO model may be a convex, bowl-likearea which includes a global minimum. This convex trust region is thenused to determine whether classifications for new, unlabeled datadeviate from a known distribution of classifications output by thetrained machine learning classifier. In some embodiments, the TRPO modelimplements importance sampling in order to reduce the number ofdirections to be monitored for the trust region. Importance sampling mayallow the meta-learning classifier to identify one or more directions inwhich multiple features have shifted within a particular time interval,which in turn may improve the efficiency of the disclosed meta-learningtechniques relative to implementation without importance sampling. Forexample, in the transaction processing context, importance sampling maytake into consideration various weighting criteria that are specific tothe fraud domain, such as weighting training examples associated withgreater potential loss (e.g., transactions associated with a largedollar amount relative to other transactions).

Multi-model cascade module 820 then uses a second model (i.e., animportance sampling model 840) to determine one or more sampling regions842 relative to the determined trust region. For example, module 820 mayuse a second model such as a proximal policy optimization (PPO) model todetermine the sampling regions 842. Module 820 may further implement athird model instead of or in addition to the second model, such as anevolve policy gradient method, to determine sampling regions 842. Insome embodiments, importance sampling includes identifying, using asecond model of the multi-model cascade process, one or more featureswhose variance satisfies a variance threshold and removing theidentified features from the trust region. Such techniques allow themulti-model cascade module 820 to identify features in the training datathat are unstable over time (e.g., features that cause greater changesin classifications relative to other features). Identification of suchfeatures may allow the retraining system 810 to reduce variance inperformance of the retraining of meta-learning classifier 830 (e.g., dueto noisy features).

That is, second and third models used in the multi-model cascade processmay further be used to reduce noisy classifications by reducing theimportance of some features (which are unstable over time) relative toother features included in training examples used to train the machinelearning classifier. This second stage of the cascade process mayfurther stabilize the performance of the meta-learning performed by themachine learning classifier. The multi-model cascade process utilizes acollection of feature selection methods using custom-defined submodularoptimization cost functions and optimizers to determine one or morefeatures that are associated with high contribution to machine learningmodel performance relative to other features. In addition, themulti-model cascade process yields a robust meta-learning classifier byaccounting for changes in underlying data over time (e.g., changes inthe underlying feature distribution used to train the machine learningmodel).

Policy module 850, in the illustrated embodiment, receives samplingregions 842 from importance sampling model(s) 840 and selects a policy852 associated with one of these regions to implement during retrainingof meta-learning classifier 830. Retraining system 810 alters classifier830 according to the selected policy 852. Once system 810 is finishedaltering classifier 830, it inputs the new set 704 of training examplesinto the altered classifier. The altered classifier 830 outputs newclassifications 834 for examples in the new set 704 based on policy 852.Retraining system 810 then uses these new classifications 834 to assignnew weights 812 to examples in the new set 704 of examples foradditional training of classifier 830. For example, as new data (withnew features) is input to classifier 830, this model needs to beretrained using updated policies to be able to accurately makepredictions for new types of data (i.e., with new features). Ifclassifier 830 were to continue implementing the same policies duringclassification of new examples with new features, the accuracy of thismodel is likely to deteriorate over time.

In various embodiments, prior to inputting examples into meta-learningmachine learning classifier 830, retraining system 810 transforms theseexamples using one or more data transformation techniques. For example,retraining system 810 may implement one or more of the datatransformation techniques discussed above with reference to FIG. 1. Insome embodiments, retraining system 810 inputs both the new set 704 ofexamples and the set 702 of testing examples into classifier 830 duringretraining. In such scenarios, retraining system 810 transforms bothsets 702 and 704 of data prior to insertion into classifier 830.Retraining system 810 may iteratively perform the retraining techniquesdiscussed herein until this system is satisfied with the output of theclassifier e.g., based on comparing the output of the model with knownlabels for the examples in the new set 704 or set 702, or both.

FIG. 8B is a diagram illustrating an example of the importance samplingperformed by the one or more importance sampling models 840 discussedabove with reference to FIG. 8A. In the illustrated embodiment, trustregion 862 is shown along a trajectory. Three different examples806A-806C of the sampling regions 842 for this trust region 862 areshown branching off into various direction from this trajectory. Notethat region 806B is along the same trajectory as the trust region 862.In this example, the solid line represents the trust region of outputgenerated by the meta-learning machine learning model, while the dottedline segments (to various regions 806) represent potential learned oradapted outputs of the meta-learning classifier after automaticretraining.

Example Auto-Labeling

FIG. 9 is a block diagram illustrating an example auto-labeling system.In the illustrated embodiment, system 900 includes meta-learning machinelearning classifier 830 and an auto-labeling system 910, which in turnincludes a comparison module 920.

Auto-labeling system 910, in the illustrated embodiment, receives knownlabels 752 for examples in new set 704. For example, computer system 710may send these examples to system 910 for determining whether theselabels have been assigned in error. Auto-labeling system 910 alsoreceives label error information 902 from an internal activity trackingsystem. As used herein, the term “label error information” includesinformation that is available from various data sources internal to thedisclosed auto-retrain system and that is indicative of whether examplesbeing used by the automatic retrain system to retrain various machinelearning models have been erroneously labeled. For example, the labelerror information might include indicators that a transaction that hasbeen labeled as credit card fraud is actually a transaction submittedfrom an account that has been hacked (i.e., an account takeover). Saidanother way, the label error information for a given example may beinconsistent with the label currently assigned this example, indicatingthat this label is wrong.

For example, this activity tracking system might observe an originallabel assigned to an example (e.g., by a system administrator) as wellas other indicators associated with this example. These other indicatorsmay indicate that this example has been mislabeled, for example. Forexample, when classifying transactions, auto-labeling system 910 mayreceive indicators based on transaction information that is availableafter a transaction has been processed. As one specific example, atransaction may be labeled as an account takeover due to suspiciousactivity on a user's credit card. A week later, however, this userreports that they lost their credit card, indicating that thistransaction should have been labeled as credit card fraud rather than anaccount takeover. Such information may be collected by an internal labelmonitoring system (e.g., a part of computer system 710) that collectsinformation associated with transactions after they have been processed.In some embodiments, this information is collected from multipledifferent sources and reported to auto-labeling system 910. For example,merchant and customer information may be monitored during transactionprocessing including their behavior history. In some situations, thisinformation may indicate that a customer account has a been taken over,credit car fraud, some form of electronic transaction fraud, or any ofvarious other types of suspicious activity.

In the illustrated embodiment, if multiple indicators are showinginconsistencies with the currently assigned label for a given example,comparison module 920 determines that the currently assigned label isincorrect. Comparison module 920 outputs one or more labels 922 that ithas determined to be incorrect. The incorrect labels 922 are thenprovided to reward/penalty module 930.

Reward/penalty module 930, in the illustrated embodiment, comparesclassifications 832 output by classifier 830 for examples in the new set704 of examples with known labels 752 for corresponding examples todetermine how much they differ. Based on the amount that they differ aswell as whether the known labels have been identified as incorrect,module 930 assigns rewards and/or penalties to these examples. Forexample, in the illustrated embodiment, module 930 assigns penalties toexamples corresponding to the incorrect labels 922 identified bycomparison module 920. These penalized examples 932 are thenbackpropagated through classifier 830 using the meta-learning techniquesdiscussed above with reference to FIGS. 8A and 8B. Once classifier 830has been retrained using the penalized examples, this classifier outputsnew corrected classifications 934 for examples in the new set 704 ofexamples.

Module 930 assigns penalties to examples based on the difference betweenthe classification output by classifier 830 and the correct label in theembodiment shown. Auto-labeling system 910 determines correct labels forincorrectly labeled examples by observing the label error information902 associated with these examples. Once system 910 has determined thecorrect labels for the mis-labeled examples, this system determinespenalties for the incorrectly labeled examples. For example, if theclassification for a particular training example is 0.9, while thecorrect label is 0, then a reward/penalty module 930 might assign apenalty of 0.9 (0.9-0) to this example (e.g., a heavy penalization). Incontrast, for correctly labeled examples, module 930 provides awards. Asone specific example, if the correct label is 1 and the classificationis 0.9, then module 930 might assign a reward of 0.1. These penalizedexamples 932 and rewarded examples are then backpropagated throughmeta-learning classifier 830 such that this classifier learns togenerate new corrected classifications 934 for these examples.Backpropagating these examples through classifier 830 retrains the modelby fine tuning the model's trust region.

These label checking techniques attempt to identify erroneously labeledexamples such that the meta-learning classifier can be retrained usingcorrectly labeled examples. In this way, the meta-learning classifierwill perform better than classifiers trained using examples whose knownlabels have not been verified. In various embodiments, retraining system810 uses the penalized (or rewarded examples) provided by auto-labelingsystem 910 to perform iterative retraining using the importance samplingtechniques discussed above with reference to FIGS. 8A and 8B. That is,auto-labeling system 910 may use the meta-learning techniques discussedabove, including the different gradient directions discussed withreference to FIG. 8B, to retrain meta-learning classifier 830 using thenew correct labels determined by auto-labeling system 910.

Consider an example in which a classification output by meta-learningclassifier 830 is 0.3. In this example, if the known label for thistransaction is 0 (indicating the transaction is not fraudulent), but thelabel error information indicates that the known label is incorrect,comparison module 920 flags this transaction as incorrectly labeled.Accordingly, reward/penalty module 930 might assign a heavy penalty tothis transaction based on the classification 0.3 being close to theincorrect label. For example, module 930 might assign a penalty of 0.7(1 minus 0.3) to this example in anticipation of the correct label being1 (indicating the transaction is fraudulent). In other situations,module 930 might assign a penalty of 0.3 divided by 0.7.

In various embodiments, the assignment of penalties and rewardsperformed by reward/penalty module 930 may be performed dynamically overtime. That is, module 930 may implement a cost function that fine-tunesassigned weights to train meta-learning classifier 830 to properlyclassify new examples whose feature distribution changes over time.

Example Methods

FIG. 10 is a flow diagram illustrating a method 1000 for automatingiterative retraining of a machine learning model, according to someembodiments. The method shown in FIG. 10 may be used in conjunction withany of the computer circuitry, systems, devices, elements, or componentsdisclosed herein, among other devices, such as those discussed belowwith reference to FIG. 12. For example, computing device 1210 is oneexample computer system that may be used to perform the disclosedtechniques. In various embodiments, some of the method elements shownmay be performed concurrently, in a different order than shown, or maybe omitted. Additional method elements may also be performed as desired.

At 1010, in the illustrated embodiment, a computer system comparesoutput of a new machine learning model for a new set of examples withknown labels for examples in the new set of examples, where the new setof examples includes one or more new features. The new machine learningmodel may be an existing machine learning classifier that has beentrained using a benchmark set of data (e.g., set 702 of testingexamples). Over time the new machine learning classifier begins todiffer from models trained using the benchmark set of data due to thedisclosed automatic and iterative retraining of this new model usingtraining examples with one or more new features. The computer system maybe any of various computing devices, servers, etc. such as thosediscussed below with reference to FIG. 12. For example, computing device1210 is one example computer system that may be used to perform thedisclosed techniques.

At 1020, the computer system determines, based on the comparing, whethera current performance of the new machine learning model satisfies aperformance threshold for machine learning models, where the performancethreshold is based on output of a benchmark machine learning model. Insome embodiments, the performance threshold is further based on adistribution summary of features included in a set of testing examplesused to test the performance of the benchmark machine learning model,where the distribution summary is generated using a generative model.The generative model might be a Gaussian process-based VariationAuto-Encoder (VAE), for example.

At 1030, in response to determining that the current performance of thenew model does not satisfy the performance threshold, the computersystem automatically triggers retraining of the new model. Theperformance of the new machine learning may be measured based on movingaverages, temporal plots, or both. If the disclosed system identifiesdeterioration of the model according to performance threshold 742, thenthe meta-learning process may be triggered. In other embodiments, theretraining process may be set to trigger periodically based on apre-defined time interval (e.g., specified by a system administrator).

In some embodiments, the determination to automatically triggerretraining of the new model is performed based on multiple differentobjectives. For example, in addition to considering whether one or morenew features have been introduced (based on the features of new examplesdiffering from the distribution summary), the computer system mayconsider: the proportion of false positives to correct classifications(e.g., a number of good transactions which the trained meta-learningclassifier incorrectly predicts as fraud), a proportion of correctlyclassified examples (e.g., true fraud that is correctly identified),temporal sensitivity (e.g., to what extent do the features of theseexamples vary over time), whether one or more new features have beenintroduced, known times of increasing attacks (e.g., increase infraudulent transactions during a holiday season), which subpopulationsof transactions (e.g., credit card, person-to-person,merchant-to-customer, etc.) suffer from the largest inaccuracies inclassifications, etc.

In some embodiments, the retraining includes determining, using a firstmodel of a multi-model cascade process, a trust region, where the trustregion is determined based on classifications output by the new modelfor the new set of examples. The first model may be a trust regionpolicy optimization (TRPO) model, for example. In some embodiments, theretraining further includes performing, using a second, different modelof the multi-model cascade process, importance sampling for one or moreregions relative to the trust region, where results of the importancesampling indicate one or more regions for altering performance of thenew model relative to a traditional gradient descent trajectory of thenew model. The second model may be a proximal policy optimizationmethod, for example, that reduces the importance of features that areunstable over time. The second model may be an evolved policy gradientmethod, for example, which may be used to lessen the variance inperformance of the meta-learning machine learning model by identifyingfeatures that are causing a threshold amount of shift in the model'sclassifications.

In some embodiments, the importance sampling includes selecting a regionfrom the one or more regions relative to the trust region and thengenerating, based on a policy associated with the selected region, newclassifications for the new set of examples. In some embodiments, theimportance sampling further includes assigning, based on the newclassifications, weights to one or more examples in the new set ofexamples. For example, as new data with new features is submitted forprocessing, the meta-learning classifier needs to be retrained to updateits policies in order to accurately make predictions for new features.

In some embodiments, the computer system determines, after theretraining, whether the known labels for examples in the new set ofexamples are correct, where the determining includes comparingclassifications output by the new model for examples in the new set ofexamples and the known labels with label error information. For example,the disclosed meta-learning techniques are performed relative to asource of truth (i.e., known labels for various examples). In order toensure that the meta-learning is as accurate as possible, the source oftruth needs to be checked after the meta-learning is complete andfurther training may be necessary if the source of truth is determinedto be wrong. For example, the meta-learning machine learning model mightoutput a classification of 0.05 for a transaction and the known labelfor this transaction is 0; however, in this example, the correct labelis 0 (the known label of 0 is identified as incorrect by the system). Inthis example, the label error information indicates that the correctlabel is 1.

In some embodiments, the label error information includes informationfrom a plurality of sources and indicates probabilities that one or moreof the known labels for examples in the new set of examples areincorrect. In some embodiments, the computer system assigns, based ondetermining that one or more known labels for examples in the new set ofexamples are incorrect, new weights to examples corresponding to the oneor more incorrect labels, where values of the new weights are determinedbased on the classifications output by the new model for examples in thenew set of examples and the label error information. For example, if aweek after a transaction is processed the label error informationindicates that the known label for a transaction is different than whatthe meta-learning classifier is predicting, then the disclosed systemassigns a penalty to this transaction prior to backpropagating itthrough the meta-learning classifier (e.g., classifier 830).

In some embodiments, the retraining includes, prior to the new modelgenerating classifications, transforming examples in the new set ofexamples and the new set of examples, where the transforming isperformed using at least one data transformation technique selectedbased on one or more types of features included in the new set ofexamples. For example, the computer system may perform z-scaling, weightof evidence, or both techniques in order to transform examples prior tobeing input into meta-learning machine learning classifier 830.

In some embodiments, the new set of examples that includes one or morenew features are recorded within a first time interval, wheredetermining the current performance of the new model is performediteratively based on a moving average of the output of the new model forexamples recorded over multiple consecutive iterations of the first timeinterval. For example, the automatic retraining techniques discussedherein may be performed iteratively based on a week-to-week movingaverage of classifications output by the meta-learning classifier over amonth. In this example, known labels are often available a week aftertransaction is initiated. Therefore, observing a week-to-week movingaverage of the meta-learning classifier's output allows the disclosedretraining system to more accurately retrain the meta-learningclassifier.

In some embodiments, the auto-labeling generates labels for examples inaddition to correcting erroneously labeled examples. That is, theauto-labeling system might generate labels for unlabeled examples basedon label error information in addition to identifying mis-labeledexamples. Such label generation may be used in place of or to augmentcurrent manual tagging processes, which in some instances areerror-prone.

Note that various examples herein classify transactions as fraudulent ornot, but these examples are discussed for purposes of explanation andare not intended to limit the scope of the present disclosure. In otherembodiments, any of various machine learning techniques as well asvarious types of classifications may be implemented.

FIG. 11 is a flow diagram illustrating a method 1100 for correctingmis-labeled examples used to train automatically retrain a trainedmachine learning model, according to some embodiments. The method shownin FIG. 11 may be used in conjunction with any of the computercircuitry, systems, devices, elements, or components disclosed herein,among other devices, such as those discussed below with reference toFIG. 12. For example, computing device 1210 is one example computersystem that may be used to perform the disclosed techniques. In variousembodiments, some of the method elements shown may be performedconcurrently, in a different order than shown, or may be omitted.Additional method elements may also be performed as desired.

At 1110, in the illustrated embodiment, a computer system performs anauto-labeling process for a set of examples used to iteratively train amachine learning model. For example, this auto-labeling process isdiscussed in further detail above with reference to FIG. 9.

At 1120, as part of the auto-labeling process, the computer systemdetermines whether known labels for examples in the set of examples arecorrect, where the determining includes comparing classifications outputby the machine learning model for examples in the set of examples andthe known labels for the set of examples with label error information.In some embodiments, the label error information includes informationfrom a plurality of sources indicating probabilities that one or more ofthe known labels for examples in the set of examples are incorrect. Inone specific example in the PAYPAL context, the label error informationmay be collected from various PAYPAL assets and data sources.

At 1130, as part of the auto-labeling process, the computer systemassigns, based on determining that one or more of the known labels areincorrect, penalties to examples corresponding to the one or moreincorrect labels. In some embodiments, the auto-labeling process furtherincludes determining, based on the label error information, correctlabels for the examples corresponding to the one or more incorrectlabels. In some embodiments, the auto-labeling process further includesbackpropagating, through the machine learning model, the penalizedexamples with corresponding determined correct labels, where thebackpropagating retrains the machine learning model such that the modeloutputs classifications corresponding to the determined correct labels.In some embodiments, the correctly labeled examples are used in futurerounds of iteratively training the machine learning model. Thereward/penalty model implemented by the auto-labeling system 910discussed above with reference to FIG. 9 may be referred to as a costfunction.

In some embodiments, values of the assigned penalties are determinedbased on the classifications output by the machine learning model forexamples in the set of examples and the label error information. In someembodiments, values of the assigned penalties are determined byidentifying, based on the label error information, correct labels forthe examples corresponding to the one or more incorrect labels. In someembodiments, the values of the assigned penalties are further determinedby determining, based on differences between the one or more incorrectlabels and the identified correct labels for corresponding examples,penalty values.

Example Computing Device

Turning now to FIG. 12, a block diagram of one embodiment of computingdevice (which may also be referred to as a computing system) 1210 isdepicted. Computing device 1210 may be used to implement variousportions of this disclosure. Computing device 1210 may be any suitabletype of device, including, but not limited to, a personal computersystem, desktop computer, laptop or notebook computer, mainframecomputer system, web server, workstation, or network computer. Note thatvarious machine learning techniques discussed herein may be performed bycomputing device 1210, for example. As shown, computing device 1210includes processing unit 1250, storage 1212, and input/output (I/O)interface 1230 coupled via an interconnect 1260 (e.g., a system bus).I/O interface 1230 may be coupled to one or more I/O devices 1240.Computing device 1210 further includes network interface 1232, which maybe coupled to network 1220 for communications with, for example, othercomputing devices.

In various embodiments, processing unit 1250 includes one or moreprocessors. In some embodiments, processing unit 1250 includes one ormore coprocessor units. In some embodiments, multiple instances ofprocessing unit 1250 may be coupled to interconnect 1260. Processingunit 1250 (or each processor within 1250) may contain a cache or otherform of on-board memory. In some embodiments, processing unit 1250 maybe implemented as a general-purpose processing unit, and in otherembodiments it may be implemented as a special purpose processing unit(e.g., an ASIC). In general, computing device 1210 is not limited to anyparticular type of processing unit or processor subsystem.

Storage subsystem 1212 is usable by processing unit 1250 (e.g., to storeinstructions executable by and data used by processing unit 1250).Storage subsystem 1212 may be implemented by any suitable type ofphysical memory media, including hard disk storage, floppy disk storage,removable disk storage, flash memory, random access memory (RAM-SRAM,EDO RAM, SDRAM, DDR SDRAM, RDRAM, etc.), ROM (PROM, EEPROM, etc.), andso on. Storage subsystem 1212 may consist solely of volatile memory, inone embodiment. Storage subsystem 1212 may store program instructionsexecutable by computing device 1210 using processing unit 1250,including program instructions executable to cause computing device 1210to implement the various techniques disclosed herein.

I/O interface 1230 may represent one or more interfaces and may be anyof various types of interfaces configured to couple to and communicatewith other devices, according to various embodiments. In one embodiment,I/O interface 1230 is a bridge chip from a front-side to one or moreback-side buses. I/O interface 1230 may be coupled to one or more I/Odevices 1240 via one or more corresponding buses or other interfaces.Examples of I/O devices include storage devices (hard disk, opticaldrive, removable flash drive, storage array, SAN, or an associatedcontroller), network interface devices, user interface devices or otherdevices (e.g., graphics, sound, etc.).

Various articles of manufacture that store instructions (and,optionally, data) executable by a computing system to implementtechniques disclosed herein are also contemplated. The computing systemmay execute the instructions using one or more processing elements. Thearticles of manufacture include non-transitory computer-readable memorymedia. The contemplated non-transitory computer-readable memory mediainclude portions of a memory subsystem of a computing device as well asstorage media or memory media such as magnetic media (e.g., disk) oroptical media (e.g., CD, DVD, and related technologies, etc.). Thenon-transitory computer-readable media may be either volatile ornonvolatile memory.

The present disclosure includes references to “embodiments,” which arenon-limiting implementations of the disclosed concepts. References to“an embodiment,” “one embodiment,” “a particular embodiment,” “someembodiments,” “various embodiments,” and the like do not necessarilyrefer to the same embodiment. A large number of possible embodiments arecontemplated, including specific embodiments described in detail, aswell as modifications or alternatives that fall within the spirit orscope of the disclosure. Not all embodiments will necessarily manifestany or all of the potential advantages described herein.

Unless stated otherwise, the specific embodiments are not intended tolimit the scope of claims that are drafted based on this disclosure tothe disclosed forms, even where only a single example is described withrespect to a particular feature. The disclosed embodiments are thusintended to be illustrative rather than restrictive, absent anystatements to the contrary. The application is intended to cover suchalternatives, modifications, and equivalents that would be apparent to aperson skilled in the art having the benefit of this disclosure.

Particular features, structures, or characteristics may be combined inany suitable manner consistent with this disclosure. The disclosure isthus intended to include any feature or combination of featuresdisclosed herein (either explicitly or implicitly), or anygeneralization thereof. Accordingly, new claims may be formulated duringprosecution of this application (or an application claiming prioritythereto) to any such combination of features. In particular, withreference to the appended claims, features from dependent claims may becombined with those of the independent claims and features fromrespective independent claims may be combined in any appropriate mannerand not merely in the specific combinations enumerated in the appendedclaims.

Because this disclosure is a legal document, various terms and phrasesmay be subject to administrative and judicial interpretation. Publicnotice is hereby given that the following paragraphs, as well asdefinitions provided throughout the disclosure, are to be used indetermining how to interpret claims that are drafted based on thisdisclosure.

References to the singular forms such “a,” “an,” and “the” are intendedto mean “one or more” unless the context clearly dictates otherwise.Reference to “an item” in a claim thus does not preclude additionalinstances of the item.

The word “may” is used herein in a permissive sense (i.e., having thepotential to, being able to) and not in a mandatory sense (i.e., must).

The terms “comprising” and “including,” and forms thereof, areopen-ended and mean “including, but not limited to.”

When the term “or” is used in this disclosure with respect to a list ofoptions, it will generally be understood to be used in the inclusivesense unless the context provides otherwise. Thus, a recitation of “x ory” is equivalent to “x or y, or both,” covering x but not y, y but notx, and both x and y. On the hand, a phrase such as “either x or y, butnot both” makes clear that “or” is being used in the exclusive sense.

A recitation of “w, x, y, or z, or any combination thereof” or “at leastone of . . . w, x, y, and z” is intended to cover all possibilitiesinvolving a single element up to the total number of elements in theset. For example, given the set [w, x, y, z], these phrasings cover anysingle element of the set (e.g., w but not x, y, or z), any two elements(e.g., w and x, but not y or z), any three elements (e.g., w, x, and y,but not z), and all four elements. The phrase “at least one of . . . w,x, y, and z” thus refers to at least one of element of the set [w, x, y,z], thereby covering all possible combinations in this list of options.This phrase is not to be interpreted to require that there is at leastone instance of w, at least one instance of x, at least one instance ofy, and at least one instance of z.

Various “labels” may proceed nouns in this disclosure. Unless contextprovides otherwise, different labels used for a feature (e.g., “firstcircuit,” “second circuit,” “particular circuit,” “given circuit,” etc.)refer to different instances of the feature. The labels “first,”“second,” and “third” when applied to a particular feature do not implyany type of ordering (e.g., spatial, temporal, logical, etc.), unlessstated otherwise.

Within this disclosure, different entities (which may variously bereferred to as “units,” “circuits,” other components, etc.) may bedescribed or claimed as “configured” to perform one or more tasks oroperations. This formulation [entity] configured to [perform one or moretasks] is used herein to refer to structure (i.e., something physical).More specifically, this formulation is used to indicate that thisstructure is arranged to perform the one or more tasks during operation.A structure can be said to be “configured to” perform some task even ifthe structure is not currently being operated. Thus, an entity describedor recited as “configured to” perform some task refers to somethingphysical, such as a device, circuit, memory storing program instructionsexecutable to implement the task, etc. This phrase is not used herein torefer to something intangible.

The term “configured to” is not intended to mean “configurable to.” Anunprogrammed FPGA, for example, would not be considered to be“configured to” perform some specific function. This unprogrammed FPGAmay be “configurable to” perform that function however.

Reciting in the appended claims that a structure is “configured to”perform one or more tasks is expressly intended not to invoke 35 U.S.C.§ 112(f) for that claim element. Should Applicant wish to invoke Section112(f) during prosecution, it will recite claim elements using the“means for” [performing a function] construct.

The phrase “based on” is used to describe one or more factors thataffect a determination. This term does not foreclose the possibilitythat additional factors may affect the determination. That is, adetermination may be solely based on specified factors or based on thespecified factors as well as other, unspecified factors. Consider thephrase “determine A based on B.” This phrase specifies that B is afactor that is used to determine A or that affects the determination ofA. This phrase does not foreclose that the determination of A may alsobe based on some other factor, such as C. This phrase is also intendedto cover an embodiment in which A is determined based solely on B. Asused herein, the phrase “based on” is synonymous with the phrase “basedat least in part on.”

The phrase “in response to” describes one or more factors that triggeran effect. This phrase does not foreclose the possibility thatadditional factors may affect or otherwise trigger the effect. That is,an effect may be solely in response to those factors, or may be inresponse to the specified factors as well as other, unspecified factors.Consider the phrase “perform A in response to B.” This phrase specifiesthat B is a factor that triggers the performance of A. This phrase doesnot foreclose that performing A may also be in response to some otherfactor, such as C. This phrase is also intended to cover an embodimentin which A is performed solely in response to B.

What is claimed is:
 1. A method, comprising: comparing, by a computersystem, output of a new machine learning model for a new set of exampleswith known labels for examples in the new set of examples, wherein thenew set of examples includes one or more new features; determining, bythe computer system based on the comparing, whether a currentperformance of the new machine learning model satisfies a performancethreshold for machine learning models, wherein the performance thresholdis based on output of a benchmark machine learning model; and inresponse to determining that the current performance of the new modeldoes not satisfy the performance threshold, automatically triggering, bythe computer system, retraining of the new model.
 2. The method of claim1, wherein the performance threshold is further based on a distributionsummary of features included in a set of testing examples used to testthe performance of the benchmark machine learning model, wherein thedistribution summary is generated using a generative model.
 3. Themethod of claim 1, wherein the retraining includes: determining, using afirst model of a multi-model cascade process, a trust region, whereinthe trust region is determined based on classifications output by thenew model for the new set of examples.
 4. The method of claim 3, whereinthe retraining further includes: performing, using a second, differentmodel of the multi-model cascade process, importance sampling for one ormore regions relative to the trust region, wherein results of theimportance sampling indicate one or more regions for alteringperformance of the new model relative to a traditional gradient descenttrajectory of the new model.
 5. The method of claim 4, wherein theimportance sampling includes: selecting a region from the one or moreregions relative to the trust region; generating, by the new model basedon a policy associated with the selected region, new classifications forthe new set of examples; and assigning, based on the newclassifications, weights to one or more examples in the new set ofexamples.
 6. The method of claim 1, further comprising: determining, bythe computer system after the retraining, whether the known labels forexamples in the new set of examples are correct, wherein the determiningincludes comparing classifications output by the new model for examplesin the new set of examples and the known labels with label errorinformation.
 7. The method of claim 6, wherein the label errorinformation includes information from a plurality of sources andindicates probabilities that one or more of the known labels forexamples in the new set of examples are incorrect.
 8. The method ofclaim 6, further comprising: assigning, based on determining that one ormore known labels for examples in the new set of examples are incorrect,new weights to examples corresponding to the one or more incorrectlabels, wherein values of the new weights are determined based on theclassifications output by the new model for examples in the new set ofexamples and the label error information.
 9. The method of claim 1,wherein the retraining includes: prior to the new model generatingclassifications, transforming examples in the new set of examples andthe new set of examples, wherein the transforming is performed using atleast one data transformation technique selected based on one or moretypes of features included in the new set of examples.
 10. The method ofclaim 1, wherein the new set of examples that includes one or more newfeatures are recorded within a first time interval, and whereindetermining the current performance of the new model is performediteratively based on a moving average of the output of the new model forexamples recorded over multiple consecutive iterations of the first timeinterval.
 11. A non-transitory computer-readable medium havinginstructions stored thereon that are executable by a computing device toperform operations comprising: determining, based on output of abenchmark machine learning model for a set of testing examples, aperformance threshold for machine learning models; comparing output of ameta-learning machine learning model for a new set of examples withknown labels for examples in the new set of examples, wherein the newset of examples includes one or more new features; determining, based onthe comparing, whether a current performance of the meta-learning modelsatisfies the performance threshold; and in response to determining thatthe current performance of the meta-learning model does not satisfy theperformance threshold, automatically triggering retraining of themeta-learning model.
 12. The non-transitory computer-readable medium ofclaim 11, wherein the one or more new features are continuous featuresthat include new ranges in values for existing continuous featuresincluded in the new set of examples.
 13. The non-transitorycomputer-readable medium of claim 11, wherein determining theperformance threshold includes: generating, by a computer system using agenerative model, a distribution summary of features included in the setof testing examples.
 14. The non-transitory computer-readable medium ofclaim 11, wherein the retraining includes: performing importancesampling for one or more regions relative to a trust region determinedusing a first model of a multi-model cascade process based onclassifications output by the meta-learning model for the new set ofexamples, wherein the importance sampling is performed using a second,different model of the multi-model cascade process, and wherein resultsof the importance sampling indicate one or more regions for alteringperformance of the meta-learning model relative to a traditionalgradient descent trajectory of the meta-learning model.
 15. Thenon-transitory computer-readable medium of claim 11, wherein theoperations further comprise: determining, after the retraining, whetherthe known labels for examples in the new set of examples are correct;and performing, based on the determining, importance sampling for one ormore regions relative to a trust region generated based onclassifications output by the meta-learning model for examples in thenew set of examples, wherein the importance sampling is performed usingat least one model of a multi-model cascade process.
 16. A method,comprising: performing, by a computer system, an auto-labeling processfor a set of examples used to iteratively train a machine learningmodel, wherein the auto-labeling process includes: determining whetherknown labels for examples in the set of examples are correct, whereinthe determining includes comparing classifications output by the machinelearning model for examples in the set of examples and the known labelsfor the set of examples with label error information; and assigning,based on determining that one or more of the known labels are incorrect,penalties to examples corresponding to the one or more incorrect labels.17. The method of claim 16, wherein the label error information includesinformation from a plurality of sources indicating probabilities thatone or more of the known labels for examples in the set of examples areincorrect.
 18. The method of claim 16, wherein the auto-labeling processfurther includes: determining, by the computer system based on the labelerror information, correct labels for the examples corresponding to theone or more incorrect labels; and backpropagating, by the computersystem through the machine learning model, the penalized examples withcorresponding determined correct labels, wherein the backpropagatingretrains the machine learning model such that the model outputsclassifications corresponding to the determined correct labels, andwherein the correctly labeled examples are used in future rounds ofiteratively training the machine learning model.
 19. The method of claim16, wherein values of the assigned penalties are determined based on theclassifications output by the machine learning model for examples in theset of examples and the label error information.
 20. The method of claim16, wherein values of the assigned penalties are determined by:identifying, based on the label error information, correct labels forthe examples corresponding to the one or more incorrect labels; anddetermining, based on differences between the one or more incorrectlabels and the identified correct labels for corresponding examples,penalty values.